Assignment 1 | CIFAR-100

https://www.kaggle.com/c/ml2016-7-cifar-100

Imports and environment setup

Set Neptune AI

Neptune example

epochs = 10
batch_size = 128
lr = 0.0001
optimizer = Adam(lr = lr)
loss = "binary_crossentropy"
 
PARAMS = {
    'epochs' : epochs,
    'batch_size' : batch_size,
    'lr' : lr,
    'optimizer' : optimizer,
    'loss' : loss,
}
tags = []
 
 
 
with neptune.create_experiment( name='VGG19',
                                params=PARAMS,
                                description='Using VGG19 model and weights to predict CIFAR-100',
                                tags=tags,
                                upload_source_files=[]) as npt_exp:
 
    callback = [
        # tf.keras.callbacks.EarlyStopping(monitor='val_loss', min_delta=0.005, patience=5, verbose=0, restore_best_weights=False)
        ModelCheckpoint('best_model.h5', monitor='val_loss', save_best_only=True),
        NeptuneMonitor(npt_exp),
    ]
 
    X_train, X_test, Y_train, Y_test = [],[],[],[]
 
    model = Model()
    model.compile(loss=loss,optimizer=optimizer, metrics=['accuracy'])
    history = model.fit(X_train, Y_train, validation_data=(X_test, Y_test), batch_size=batch_size,
                        epochs=epochs,
                        verbose=1, callbacks=callback)
 
    results = model.evaluate(X_test, Y_test, batch_size=batch_size)
    names = model.metrics_names
    npt_exp.set_property(names[0], results[0])
    npt_exp.set_property(names[1], results[1])
 
    model.save_weights('model_weights.h5')
    npt_exp.log_artifact('model_weights.h5')
    npt_exp.log_artifact('best_model.h5')

Load datasets

1. Data exploration

a. Data size

As we can see above, Train set has 50,000 records and the Test set has 10,000 records.

Number of unique targets is 100

b. What data does each sample contain?

Each sample contains a simgle Image and its classification

Let's take a look at some examples from the Trainset

Dimensions, channels, how many classes

• Dimensions: (32x32) - each image has 32x32 pixels.
• Channels: 3 that represents the RGB colors
• Number of classes: As mentioned before, the num of classes is 100

Should we preprocess the data?

We are using data from the Keras.datasets, thus the data is ready for use.

Can we use augmentation? what kind would be valid?

We can use augmentation, rotations, movements and croping that can be applied here.

But as we can see, the given images are well croped and the main element is centered therefor there is no need for augmentation for this dataset.

Edit - Later while working on the model's improvements, found out that augmentation is helpful (more details at 2.d)

c. Class distribution - is it balanced?

The data is perfectly balanced - Each target has the exact same number of records ( 500 for the train, 100 for the test )

d. Are there any benchmark results for different methods used on this data?

https://paperswithcode.com/sota/image-classification-on-cifar-100

We can see that the top 4 methods were able to get an accuracy score above 90%

Note that most of them were using 'Extra training data', this was not common in previous methods ( from #6 downwards )

e. Show some samples for each lable ( easy vs hard )

Hard separables might be caused by similar backgrounds and\or colors of elements in the pictures, as well as similar shapes of the objects.

Easy separables can be seen as images with different structures ( colors, texture, backgrounds)

2. Build model

Save and Load models

BuildModel

Input shape

  4D tensor with shape:
  if `data_format` is `"channels_first"` - `(batch, channels, rows, cols)`
  if `data_format` is `"channels_last"` - `(batch, rows, cols, channels)`

a. Validation strategy for traning the model

We use validation_split of 0.2

Train size will be 80%, Test size is 20%

Loading model...

Model's Prediction

Confusion matrix

Present loss and other metrics

Show examples for good and bad classification with high probability and refer to the uncertain predictions

By looking at the examples, even humans might find it difficult to classify the first row. Trees look alike and it's hard to get the correct one (same for Boy-Man and Squirrel-Rabbit)

The second row doesn't have any elements that might be similar between expected and actual prediction

Compare testset result with the validation

Accuracy on testset was very similar to the validation accuracy, even a bit better

On validation: 36% | On testset: 38.7%

c. How to improve?

https://www.machinecurve.com/index.php/2020/03/30/how-to-use-conv2d-with-keras/

loss_function = sparse_categorical_crossentropy

List at least 3 ways to improve

1. Normalize the data, using X_train.astype('float32')/255
2. Build Conv2D layers up to size 256 (instead of 64)
3. Use more epochs (instead of 10)
4. Add more data to training by changing thr validation split ratio
5. Using Augmentation

Prioritize improvements in a list

d. Implement the first 2 improvements

In the first atempt to improve we will change the following

• Add more Conv2D layers, up to 256
• Normalize the input data so it's values are between 0 to 1
• Add more epochs to fit the model (we will use 50 epochs)

Build improved model

Confusion matrix - improved

Improve Using augmentation

https://keras.io/preprocessing/image/

Example here: https://www.youtube.com/watch?v=14syUbL16k4

Improve graph

As we can see, even though the improved model got better results - it's still overfitting

The red line represents the basic_model's validation status

In order to overcome the overfitting, we will try to add more samples to the training data

We will do so by setting the validation_split to 0.9

In addition, we add ImageDataGenerator for augmentation usage

Augmentation parameters

1. rotation_range=10 ( Image will rotate at most 10 deg )
2. width_shift_range=0.1 ( Image will shift horizontally )
3. height_shift_range=0.1 ( Image will shift vertically )
4. shear_range=0.01 ( Shear angle in counter-clockwise direction in degrees)
5. zoom_range = 0.05 ( Image will be zoomed-in )
6. horizontal_flip=True ( Allowing horizontal flip )
7. vertical_flip=False ( Vertical flip is not allowed )

Augmentation examples

Training the model